An Examination of the Congruence of Literacy 1 



Running head: AN EXAMINATION OF THE CONGRUENCE OF LITERACY 



An Examination of the Congruence of Literacy Instruction from Middle Schools to High Schools 

Beverly M. Klecker 
Mary Anne Pollock 
Morehead State University 
Morehead, Kentucky 
b . klecker @ morehead- st.edu 



Paper presented at the annual meeting of the 
Mid-South Educational Research Association 
Gatlinburg, TN 



November 19, 2004 




An Examination of the Congruence of Literacy 2 



Abstract 

Kentucky’s goal of reaching academic "proficiency" by 2014 illuminated problematic findings in 
2002 reading test scores: 44.30% of middle and 71.25% of high school students scored below 
“proficient.” The research question was, "Do teaching practices in schools with high reading 
achievement scores differ from teaching practices in schools with low reading achievement 
scores?" High school reading scores were stratified by region (2) then high/low. Five high 
schools were randomly selected from each strata (N=20). Feeder schools were added (N=19). 

450 of 638 (70.5%) eighth through 10th grade teachers within these schools completed surveys 
in April 2003. A random sample of five of the teachers who had completed the surveys within 
each school was selected. These teachers were first observed then interviewed. Structured 
instruments were used for both the observations and interviews. Lastly, the Comprehensive 
School Improvement Plan for each school was analyzed to identify strategies to improve reading 
within each school. Differences found through each strategy are explained and compared. 
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An Examination of the Congruence of Literacy Instruction 
from Middle Schools to High Schools 

Reading is assessed by the Kentucky Core Content Tests (KCCT) at the middle school 
level in the 8th grade and at the high school level in the 10th grade. Framing the following data 
with Kentucky’s goal of getting to proficiency by 2014 illuminated the problem. Statewide, 
44.30% of the middle school students’ 2002 KCCT Reading scores were below “Proficient.” 
Alarmingly, this jumped to 71.25% of high school students’ Reading score falling below 
“Proficient.” (Kentucky Department of Education). Since teachers certified to teach in middle 
schools are required to complete at least one course in reading pedagogy and high school 
teachers are not required to complete any courses in reading pedagogy, high school teachers may 
not be using research-based reading strategies in their classrooms. 

The research questions guiding this study were: 

1. Do teaching practices in schools with high reading achievement scores differ from 
teaching practices in schools with low reading achievement scores? 

2. Are there differences in the inclusion of literacy goals in the Comprehensive 
School Improvement Plan between schools with high reading achievement scores 
and schools with low reading achievement scores? 

Methods 

Sampling Plan 

A stratified random sampling plan was used to identify school districts in two geographic 
areas of the state: (1) Western Kentucky, defined as the school districts in Kentucky's Regional 
Service Centers 1 and 2; and (2) Eastern Kentucky, defined as the school districts in Kentucky's 



Regional Service Centers 7 and 8. 
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The 2002 10th grade Kentucky Core Content Reading scores were stratified first by 
Western and Eastern Kentucky. The scores were then sorted from highest to lowest within each 
of the two geographical regions. The 25 highest scoring schools and 25 lowest scoring schools 
within each region served as the target sample group. A random sample of five (5) high schools 
was selected from each of the four strata. The 20 high schools, with all of the schools identified 
as their feeder schools constituted the randomly selected sample for this study. A feeder school 
was identified as any school, which sends students on to a high school; this includes middle 
schools as well as elementary schools, which contain grades 7 and 8. Three of the 20 high 
schools are K- 12 schools so have no feeder schools; one high school has three feeder schools, 
and the remaining 16 schools have middle schools which feed into them. A total of 39 schools 
(20 high schools, 16 middle schools, and three K-8) participated in the study. Only classrooms 
in grades 8, 9, and 10, within the high schools and feeder schools were included in the study. 
After the 20 high schools were selected through the sampling technique described above, the 
Kentucky School Directory was used to identify their feeder schools. 

Procedure 

After the high schools and their feeder schools had been identified, letters describing the 
study and asking for permission to participate in the study were sent to district superintendents. 
All of the superintendents granted permission for the principals of identified schools to be 
contacted. Principals were contacted to obtain permission to participate; however, four of the 
high school principals chose not to participate in the study. To replace these schools, the 
Principal Research used the original stratified sampling frame and the same random number to 
randomly select replacement schools. Superintendents of the replacement schools were contacted 
as described above, and then principals were contacted; all agreed to participate in the study. 
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The researchers traveled to the 39 schools to administer the survey to teachers during 
faculty meetings. The researchers described the study and invited teachers of the 8th, 9th, and 
10th grade students to participate through surveys, classroom observations, and interviews. A 
signed consent form to participate in the study was obtained from teachers; only teaches who 
signed the consent form participated in the study. 

Measurement Instruments . 

The validity of the instruments used in this study was established through review by two 
professors who are experts in the field of reading to verify that they are based on current 
research. Both experts, who were not members of the research team, agreed that the instruments 
were indeed based on current research. 

Training of raters. 

Members of the research team from Morehead State University, Pikeville College, and 
Murray State University met in Elizabethtown, Kentucky for two days in July, 2003 to be trained 
in classroom observation, interview procedures, and analysis if the Comprehensive School 
Improvement Plan (CSIP). They used the Teacher Observation in Content Area Lesson 
Instrument as they watched video clips of classroom scenarios to assure that they would all be 
responding in like manner during classroom observations. This training resulted in an inter-rater 
reliability of for the Teacher Observation in Content Area Lesson Instrument r xx =. 83. 

Data Collection, Data Analysis, and Results 

This section of the report is organized in four sub-sections around the four instruments: 
(1) Teacher Survey My Practices in Content Area Lessons , (2) Teacher Obserx’ation in Content 
Area Lessons , (3) Teacher Interx’iew, and (4) The Comprehensive School Improvement Plan 
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(CSIP) (see Appendix). The presentation within each of the four sections includes descriptions of 
the data collection, data analysis, and results. 

Teacher Survey 

Data collection with teacher survey. 

In the spring of 2003, the teacher survey was distributed to all eighth through tenth grade 
teachers in the 39 schools in the sample. Members of the research team visited the schools and 
collected the data at a regularly scheduled teachers' meeting. Six hundred and fifty-six forms 
were returned. Of these 656 forms, 180 were blank, resulting in 476 completed survey forms. 
Four hundred and fifty forms were usable, that is, the consent form for the teacher was present 
and the data were complete. (No data were received from one middle school, thus the N for the 
school level analyses is 38.) Table 1 presents the number of responding teachers by content area 
within these 38 schools. Table 2 presents the grade levels taught by teachers responding to the 
surveys. From the data in these two tables, it is clear that the study was truly across content areas 
and included teachers in grades eight through ten. 
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Table 1. Content Areas Taught by Teachers Responding to the Teacher Survey 



Content Area(s) Taught 


N 


Foreign Language 


13 


Science 


45 


Arts & Humanities 


16 


Mathematics 


73 


Social Studies 


63 


English/Language Arts 


80 


Health/Physical Education 


16 


Special Education 


26 


Agriculture 


6 


Business Education 


7 


Family/Consumer Science 


10 


Practical Living 


5 


Technical Education 


11 


Reading 


8 


Music 


9 


Welding 


1 


Media Specialist 


1 


Not Identified 


60 


Total 


450 



Table 2. Grade Level(s) Taught by Teachers Responding to the Teacher Survey 



Grade Level(s) Taught 


N 


Eighth Grade 


115 


Eighth and Ninth Grade 


2 


Eighth, Ninth, and Tenth Grade 


15 


Ninth Grade 


41 


Ninth and Tenth Grade 


111 


Tenth Grade 


60 


Not Identified 


40 


Total 


450 
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Data analysis for teacher survey. 

The responses of 450 teachers to Teacher Survey were aggregated by school— the unit of 
analysis for this study. The number of teacher responses within schools ranged from two to 45. 
The mode was seven per school, the Teacher Survey data were entered into an Excel spreadsheet 
and uploaded into Statistical Analysis Software (SAS) for analysis on a PC. Cronbach's alpha 
reliability was calculated for the Teacher Survey using the data from the study. The internal 
consistency reliability was 0.85. Means and standard deviations were then calculated from the 
data collected five-point Likert-type item rating scales for the 20 items of the survey. 
Independent sample Mcsts were used to test for statistically significant (p<.05) differences 
between the measures for the High Scoring High Schools and Low Scoring Schools (as defined 
by scores on the 2002 KCCT Reading tests). 

Results of teacher survey analysis. 

The data for the Teacher Survey are presented in Table 3. The Likert-type, five-point, 
item rating scale ranged from l="Not at all," 3="To some extent,” to 5="A great deal." The 
differences in the means in Table 3 are all of degree rather than of kind; that is, all of the mean 
ratings are above the 3.00 "To some extent" rating. The mean ratings for the High Scoring 
Schools (N=22) in Table 3 indicate that the highest rated item by teachers within these schools 
was Item 12, "Students increase their knowledge by responding to questions either orally or in 
writing" (M=4.58, SD=0.3 1 . The item with the highest mean rating in the Low Scoring Schools 
(N=16) was Item 8, "I give students a specific task to accomplish during the lesson" (M=4.74, 



SD= 0.28). 
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Table 3. Responses to Teacher Survey Items by High Scoring and Low Scoring Schools 



Teacher Survey Item 


High Scoring 
Schools 
N=22 


Low Scoring 
Schools 
N=16 


Test for Statistically 
Significant Differences 




Mean 


SD 


Mean 


SD 


t value 


p value 


1 . I alter the list of vocabulary words provided by the textbook. 


3.39 


0.75 


3.05 


0.50 


1.519 


p= 0.1147 


2. I take time to develop vocabulary at the beginning of the lesson. 


3.79 


0.50 


3.90 


0.48 


-0.744 


p=0.4578 


3. The vocabulary strategies I use actively involve students. 


3.83 


0.59 


3.84 


0.41 


-0.068 


p=0.9459 


4. I think about the degree and accuracy of prior knowledge my 
students have before planning the lesson. 


4.24 


0.50 


4.17 


0.49 


0.389 


p= 0.6993 


5. I choose a pre-lesson strategy based on the students' prior knowledge. 


3.73 


0.43 


3.66 


0.49 


0.418 


p=0.6786 


6. I involve students in some type of pre-lesson strategy. 


3.58 


0.53 


3.61 


0.57 


-0.188 


p= 0.8520 


7. I establish a puipose or motivation for learning before students are asked to 
read/learn. 


4.20 


0.50 


4.17 


0.40 


0.231 


p=0.8121 


8. I give students a specific task to accomplish during the lesson. 


4.51 


0.50 


4.74 


0.28 


-1.816 


p= 0.0782 


9. I provide students with a strategy to keep them actively involved during the 
lesson. 


4.26 


0.40 


4.30 


0.37 


-0.378 


p= 0.7075 


10. 1 loop discussion back to pre-lesson activities. 


4.27 


0.49 


4.12 


0.50 


0.894 


p-0.3110 


1 1. 1 encourage students to increase their knowledge by sharing. 


4.26 


0.42 


4.09 


0.48 


1.126 


p= 0.2677 


12. Students increase their knowledge by responding to questions either orally 
or in writing. 


4.58 


0.31 


4.59 


0.37 


-0.172 


p=0.8643 


13. Students increase their knowledge by consulting texts or other sources. 


4.13 


0.37 


4.14 


0.43 


-0.108 


p=0.9148 


14. 1 provide the opportunity for students to reflect on their learning. 


3.90 


0.64 


3.97 


0.38 


-0.384 


p= 0.7036 


15.1 provide the opportunity for students to reflect on the effectiveness of 
any strategies they used in the lesson. 


3.52 


0.66 


3.38 


0.50 


0.720 


p=0.4761 


16. 1 involve students in writing at some point during the lesson. 


4.08 


0.57 


3.87 


0.40 


1.234 


p=0.2251 


17 1 use grouping (pairs to small groups) successfully to engage students 
in learning. 


4.01 


0.35 


3.59 


0.51 


2.973 


p=0.0052* 


18. 1 engage students in using context clues for the vocabulary words at some 
point during the lesson. 


3.76 


0.50 


3.49 


0.48 


0.828 


p=0.4130 


19. 1 engage students in using context clues to guide their reading. 


3.78 


0.51 


3.59 


0.65 


1.054 


p=0.2991 


20. 1 read aloud to students from a variety of sources. 


3.72 


0.57 


3.72 


0.66 


-0.022 


p=0.9830 



Note: * Statistically significant p<.05 
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The item with the lowest mean rating by the High Scoring Schools was Item 1, "I alter 
the list of vocabulary words provided by the textbook" (M= 3.39, SD=0.15). This was also the 
item with the lowest mean rating by the Low Scoring Schools (M=3.05, 5D=0.50). 

This finding was indicative of the findings from the data presented in Table 3. There were not 
large differences in the mean ratings of the survey items by school achievement level. 

Statistically significant difference (p<.05). 

There was a statistically significant difference (p<. 05) between the ratings of school 
achievement level groups on only one item, Item 17: "I use grouping (pairs to small groups) 
successfully to engage students in learning." On this item, the mean rating for the High Scoring 
Schools (M=4.01, SD= 0.35) was statistically significantly higher than the mean rating for the 
Low Scoring Schools (M= 3.59, 50=0.51) [t (36)=2.793, p<.05]. 

Teacher Observation 

Data collection with teacher observations. 

A random sample of five teachers was selected from each of the schools from which 
Teacher Survey data were obtained. Members of the research team contacted the teachers and 
arranged for (1) an observation of the teacher's classroom, and (2) a follow-up interview after the 
observation. Structured forms were used for both the observation and the interview. The teacher 
observations and interviews were conducted from fall 2003 through spring 2004. 

Despite the research design, real-life events including school scheduling, faculty attrition, 
and weather realities limited the number of observations that the team was able to complete. 

• In nine of the 38 schools, five teacher observations were completed (N=45). 

• In six of the 38 schools, four teacher observations were completed per school 



(N=24). 
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• In seven of the 38 schools, three teacher observations were completed per school 
(N=21). 

• In ten of the 38 schools, two observations were completed per school (N=20). 

• In six of the 38 schools, only one teacher observation was completed in each 
school (N=6). Thus, from the target sample of five teachers in each of 38 schools 
(N=190), 116 teacher observations in 38 schools were completed (61% of the 
target sample). 

Table 4 presents the number of observed teachers by content area within these 38 
schools. Table 5 presents the grade levels taught by observed teachers. From the data in these 
two tables, it is clear that the study was truly cross content areas and included teachers in grades 
eight through ten. 

Table 4. Content Areas Taught by Teachers Observed in Their Classrooms 



Content Area(s) Taught 


N 


Foreign Language 


4 


Science 


17 


Arts & Humanities 


3 


Mathematics 


21 


Social Studies 


21 


English/Language Arts 


26 


Health/Physical Education 


5 


Special Education 


2 


Agriculture 


2 


Business Education 


1 


Family/Consumer Science 


3 


Practical Living 


1 


Technical Education 


1 


Reading 


4 


Music 


1 


Not Identified 


4 


Total 


116 
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Table 5. Grade Level(s) Taught by Teachers Obserx’ed in Their Classrooms 



Grade Level(s) Taught 


N 


Eighth Grade 


50 


Ninth Grade 


20 


Ninth and Tenth Grade 


20 


Tenth Grade 


18 


Not Identified 


8 


Total 


116 



Data analysis for teacher obserx’ations. 

The observation instrument. Teacher Observation in Content Area Lesson (see 
Appendix), contained 20 items. Each item had a three point rating scale. This rating scale was 
categorical, l="No, I did not observe;" 2="Not sure;" and, 3="Yes, I did observe." Space was 
provided for the observer to record "Comments." The data from the 1 16 teacher observations 
were entered into an Excel spreadsheet. They were then aggregated by school by using the modal 
observation (most frequent value) within each school. The "Not sure" category was selected very 
infrequently by the observers. Because this category added no meaning to the analysis, it was 
dropped before proceeding with the data analysis. The categorical school-level data were then 
uploaded into SAS in a PC and were analyzed using frequencies and percentages. Chi-square 
tests were used to compare the "No, I did not observe" and "Yes, I did observe" categorical 
frequencies and percentages with the High Scoring Schools and Low Scoring Schools categories. 
Alpha level was set a priori at .05. 

Results of teacher observation analysis. 

Table 6 presents the 20 items of the teacher observation instrument with the total 
percentage observed overall, the percentage observed in the High Scoring Schools, the 
percentage observed in the Low Scoring Schools, the computed Chi-square statistic, and the p 
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value. Items 6, 7, and 10 were the most frequently observed (87%) in the total sample of schools. 
The least observed practice in the total sample of schools was Item 17, "Teacher engages 
students in using context clues to guide their reading." This was observed in only 28% of the 
schools in the overall sample. 

Descriptive differences within the High Scoring Schools. 

The most frequently observed strategies in the High Scoring Schools were Items 6, 
"Teacher gives students a specific task to accomplish during the lesson" and Item 7, "Teacher 
provides students with a strategy/activity to keep them actively involved during the lesson;" 
each was observed in 90% of the schools. The least frequently observed strategies in the High 
Scoring Schools, each observed in 19% of the schools, were Item 13, "Teacher provided the 
opportunity for students to reflect on the effectiveness of any strategies they used in the lesson" 
and Item 17, "Teacher engages students in using context clues to guide their reading." 

Descriptive differences within the Low Scoring Schools. 

The most frequently observed strategies in the Low Scoring Schools (observed in 88% of 
the schools) were Item 5, "Teacher establishes a purpose or motivation for learning before 
students are asked to read/leam;" Item 10, "Students increase their knowledge by responding to 
questions either orally or in writing;" and Item 14, "Teacher involves students in writing at some 
point during the lesson." The least frequently observed practice (35%) in the Low Scoring 
Schools was Item 15, "Teacher uses group (pairs to small groups) successfully to engage 
students in learning." 

Statistically significant differences (p<.05). 

In two of the observed items, there were statistically significant (p<.05 ) differences 
between the observed frequencies of teacher practices in the High Scoring Schools and the Low 
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Scoring Schools. Item 1, "Teacher takes time to develop vocabulary at the beginning of the 
lesson," was observed more frequently (76%) in the Low Scoring Schools than in the High 
Scoring Schools (43%) [A 2 (1 )=4.35,/?<.05J. Additionally, Item 16, "Teacher engages students in 
using context clues for the vocabulary words at some point during the lesson" was observed 
statistically significantly (p<.05) more frequently in Low Scoring Schools (65%) than in High 
Scoring Schools (24%) [A a (l)=6.45, p<.05]. 




An Examination of the Congruence of Literacy 



15 



Table 6. Percentages of Teacher Strategies Observed in High Scoring and Low Scoring Schools 



Observation Instrument Item 


% Obsvd 
All 
N=38 


%Obsvd 
H Score 
N=21 


%Obsrd 
L Score 
N=17 


Chi- 

Square 


p value 


1. Teacher takes time to develop vocabulary at the beginning of the lesson. 


58% 


43% 


76% 


4.35 


p= 0.0369* 


2. Teacher uses vocabulary development strategies/activities that actively 
involve students. 


45% 


33% 


59% 


2.47 


/?=0. 1161 


3. Teacher chooses a pre-reading activity based on the students' 
knowledge. 


50% 


38% 


65% 


2.66 


p=0.1028 


4. Teacher involves students in some type of pre-lesson strategy. 


61% 


47% 


76% 


3.27 


p=0.0704 


5. Teacher establishes a puipose or motivation for learning before 
students are asked to read/learn. 


82% 


76% 


88% 


0.91 


p=0.3409 


6. Teacher gives students a specific task to accomplish during the lesson. 


87% 


90% 


82% 


0.54 


p=0.4614 


7. Teacher provides students with a strategy/activity to keep them actively 
involved during the lesson. 


87% 


90% 


82% 


0.54 


p=0.4614 


8 Teacher loops discussion back to pre-lesson activities. 


55% 


62% 


47% 


0.84 


p=0.3601 


9. Teacher encourages students to increase their knowledge by sharing. 


74% 


67% 


82% 


1.19 


p=0.2749 


10. Students increase their knowledge by responding to questions either 
orally or in writing. 


87% 


86% 


88% 


0.05 


p=0.8192 


11. Students increase their knowledge by consulting texts or other sources. 


74% 


67% 


82% 


1.19 


p=0.2749 


12. Teacher provided the opportunity for students to reflect on their 
learning. 


50% 


38% 


65% 


2.66 


p=0.1028 


13. Teacher provided the opportunity for students to reflect on the 
effectiveness of any strategics they used in the lesson. 


32% 


19% 


47% 


3.41 


p=0.0647 


14. Teacher involves students in writing at some point during the lesson. 


82% 


76% 


88% 


0.90 


p=0.3409 


15. Teacher uses group (pairs to small groups) successfully to engage 
students in learning. 


37% 


38% 


35% 


0.03 


p=0.8587 


16. Teacher engages students in using context clues for the vocabulary 
words at some point during the lesson. 


42% 


24% 


65% 


6.45 


p=0.0111* 


17. Teacher engages students in using context clues to guide their reading. 


28% 


19% 


41% 


2.23 


p=0.1348 


18. Teacher reads aloud to students. 


55% 


48% 


65% 


1.11 


p= 0.2922 


19. Teacher asks students to read aloud. 


50% 


48% 


53% 


0.10 


p=0.7475 


20. Teacher uses technology during the lesson. 


66% 


67% 


65% 


0.02 


P= 0.8992 



*Note: p values in bold significant .05 
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Teacher Interviews 

Data collection for teacher interx’iews. 

Following the teacher observation, the observer interviewed the teacher using the 
structured Teacher Interview form (see Appendix). One hundred and sixteen teachers were 
interviewed, 59 taught in the High Scoring Schools and 57 taught in the Low Scoring Schools. 

Qualitative data analysis of teacher interviews. 

The unit of analysis for the qualitative data was the teacher as it was not realistic to 
aggregate the statements of individual teachers to the school level. The interview data were 
entered into a database in Microsoft Word format. Statements were coded by item number and 
by teacher ID number. The statements were then sorted by High Scoring Schools and Low 
Scoring Schools for each of the 25 items of the Teacher Interview. The statements were then 
coded by emerging categories within the items. Frequencies and percentages were calculated for 
the emergent categories within each question. Differences were noted by comparing the 
percentage of teachers making the statement within the High Scoring Schools and the Low 
Scoring Schools. When examining Table 7, it is important to note that a teacher's single 
statement often contained more than one category mentioned. Thus, the Ns cannot be easily 
interpreted. They are presented to allow the calculation of the frequencies and percentages of the 
mention of the categories. The highest percentages for each question for both the High Scoring 
and the Low Scoring Schools have been shaded with a gray-tone highlight. 
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Table 7 (A). Teacher Interview Results by High and Low Scoring Schools (Qualitative Data Analysis) 



Question 


High Scoring 
Schools 


Low Scoring 
Schools 




N 


% 


N 


% 


1. Where did you learn the classroom practices you used today? 










• College or university 


36 


31% 


30 


31% 


• Professional development 


20 


17% 


18 


19% 


• Experience 


14 


12% 


18 


19% 


• Mentors 


14 


12% 


10 


11% 


• Trial and Error 


11 


9% 


4 


4% 


• Professional literature 


8 


7% 


5 


5% 


• Professional organizations/conferences 


6 


5% 


0 


0% 


• Observations 


4 


3% 


2 


2% 


• Internet 


1 


1% 


4 


4% 


• Reflection 


1 


1% 


3 


3% 


• Other 


3 


2% 


1 


1% 


Totals for Question 1 


118 


100% 


95 


100% 












2. How does this lesson compare to a typical day? 










• Consistent with a typical day 


33 


51% 


35 


67% 


• Not typical 


12 


19% 


8 


15% 


• Not responsive to question 


19 


30% 


9 


17% 


Totals for Question 2 


64 


100% 


52 


100% 












3. How does this lesson relate to a previous lesson? Describe 










• Sequence implied 


56 


88% 


42 


81% 


• No sequence implied 


5 


8% 


7 


13% 


• Starting a new unit 


2 


3% 


3 


6% 


• Not responsive to question 


1 


1% 


0 


0% 


Totals for Question 3 


64 


100% 


52 


100% 
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Table 7. (B) Teacher Interview Results by High and Low Scoring Schools (Qualitative Data Analysis) 



Question 


High Scoring 
Schools 


Low Scoring 
Schools 




N 


% 


N 


% 


4. Tell me about some specific strategies you use in your classroom. 










• Mentioned instructional practices that are not really strategies 


33 


51% 


33 


65% 


• Specific strategies stated 


21 


32% 


11 


21% 


• Hinted at strategies but not stated 


8 


12% 


5 


10% 


• Classroom instruction/management 


1 


1% 


2 


4% 


• Doesn't use/could not name 


2 


3% 


0 


0% 


• Not responsive to question 


1 


1% 


0 


0% 


Totals for Question 4 


66 


100% 


51 


100% 












5. Which strategies do you observe your students using independently? 










• Identified instructional activities that are not strategies 


44 


64% 


32 


68% 


• Identified strategies students use independently 


12 


17% 


10 


21% 


• Indicated that students don't use strategies independently 


10 


15% 


2 


4% 


• Not responsive to question 


3 


4% 


3 


6% 


Totals for Question 5 


69 


100% 


47 


100% 












6. How do you introduce a new lesson to your students? 










• The fact that teacher accounts for prior knowledge is directly stated. 


17 


16% 


10 


13% 


• Teacher accounts for prior knowledge. 


4 


4% 


10 


13% 


• Preview/overview of lesson given 


9 


8% 


7 


8% 


• Specific strategies are stated. 


7 


7% 


1 


1% 


• Relate to real life/personal experiences 


10 


9% 


8 


10% 


• Teacher uses hook/anticipatory set/p rompt/bell ring or other focusing activities. 


16 


15% 


2 


3% 


• Vocabulary is addressed 


11 


10% 


10 


13% 


• Essentials/objectives/other questions used 


7 


7% 


8 


10% 


• Discussion 


7 


7% 


3 


4% 


• Prewriting in journals/other prewriting 


4 


4% 


1 


2% 


• Handouts/worksheets/study guides 


2 


2% 


1 


2% 


• Teacher tells students why they need to learn/how it will be helpful to know 


1 


1% 


3 


4% 


• Skills are developed 


2 


2% 


1 


2% 





An Examination of the Congruence of Literacy 



19 



Table 7. (C) Teacher Interview Results by High and Low Scoring Schools (Qualitative Data Analysis) 



Question 


High Scoring 
Schools 


Low Scoring 
Schools 


6. How do you introduce a new lesson to your students? (Continued) 


N 


% 


N 


% 


• Read/discuss/answer 


2 


2% 


4 


5% 


• Review previous lesson 


2 


2% 


4 


5% 


• Lecture 


0 


0% 


2 


3% 


• Not responsive to question 


4 


4% 


3 


4% 


Totals for Question 6 


105 


100% 


74 


100% 












7. What instructional materials do you use? 










• Textbooks 


47 


22% 


38 


24% 


• Newspapers/journals/articles/magazines/flyers/brochures 


40 


18% 


26 


17% 


• Internet/other technology 


37 


17% 


26 


17% 


• Resource Material: History Alive, library, supplemental books 


24 


11% 


9 


6% 


• Literature: plays, poems, novels, picture books 


20 


9% 


20 


13% 


• Teaching tools, dice, maps, music, dictionary 


19 


9% 


10 


6% 


• Books on tape, films/videos, TV 


12 


5% 


15 


9% 


• Handouts, workbooks, notebooks, worksheets 


12 


5% 


12 


7% 


• Limited/no use of textbooks 


9 


4% 


2 


1% 


Totals for Question 7 


220 


100% 


158 


100% 












8. Tell me how you integrate reading into your content area. Give examples. 










By having my students read: 










• Periodicals, magazines, newspapers, journals, brochures 


30 


28% 


19 


25% 


• Narrative texts: novels, plays, poems, trade books 


23 


22% 


16 


21% 


• The assignment/chapter 


18 


17% 


20 


28% 


• Technology: Internet, software, PowerPoint, Web Quests 


10 


9% 


5 


7% 


• Accelerated Reader books 


3 


3% 


0 


0% 


• Procedural text: directions, labs, instructions 


4 


4% 


7 


9% 


• Expository material other than the textbook 


4 


4% 


6 


8% 


• Mentioned lesson organization, and instructional activities but not materials. 


8 


8% 


1 


1% 


• Everything is reading/English so it is in everything 


5 


5% 


1 


1% 


Totals for Question 8 


105 


100% 


75 


100% 
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Table 7. (D) Teacher Interview Results by High and Low Scoring Schools (Qualitative Data Analysis) 



Question 


High Scoring 
Schools 


Low Scoring 
Schools 


9. Are your lessons structured to incorporate some reading each day? For 










example, reading lab, instructions, directions, etc. Give examples. 


N 


% 


N 


% 


• Yes 


55 


85% 


45 


80% 


• No 


6 


9% 


8 


14% 


• Most of the time but not everyday 


4 


6% 


1 


2% 


• Not responsive to question 


0 


0% 


2 


4% 


Totals for Question 9 


65 


100% 


56 


100% 












10. Do you read aloud to your students? 










• Yes 


56 


86% 


49 


90% 


• Sometimes 


3 


5% 


3 


5% 


• No 


6 


9% 


3 


5% 


Totals for Question 10 


65 


100% 


55 


100% 












11. What do you read to them? 










• Textbook 


29 


26% 


29 


32% 


• Novels/other literature 


25 


23% 


20 


23% 


• Articles/newspapers/maps 


20 


18% 


15 


16% 


• Instructional procedures (directions, etc.) 


19 


17% 


16 


18% 


• Expository texts other than textbook 


11 


10% 


5 


5% 


• Technology/e-mail 


0 


0% 


2 


2% 


• Students' writing 


2 


2% 


0 


0% 


• Teachers’ writing 


2 


2% 


1 


1% 


• Does not read to students 


1 


1% 


1 


1% 


• Not responsive to question 


1 


1% 


2 


2% 


Totals for Question 11 


110 


100% 


91 


100% 
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Table 7. (E) Teacher Interview Results by High and Low Scoring Schools (Qualitative Data Analysis) 



Question 


High Scoring 
Schools 


Low Scoring 
Schools 


12. Please describe the various ways you use this practice. 










• To assure students comprehend 


24 


41% 


13 


23% 


• To share new information/to inform 


10 


17% 


17 


30% 


• To model 


10 


17% 


5 


9% 


• To motivate 


4 


6% 


4 


7% 


• To emphasize a point 


3 


5% 


2 


4% 


• To entertain 


1 


2% 


3 


5% 


• To support poor readers 


1 


2% 


5 


9% 


• Not responsive to question 


6 


10% 


8 


13% 


Totals for Question 12 


59 


100% 


57 


100% 












13. How do your students benefit from this practice? 










• Helps them understand 


30 


39% 


22 


47% 


• Skill related/builds skills 


13 


17% 


10 


21% 


• Motivates/creates interest for pleasure 


10 


14% 


3 


6% 


• For pleasure 


5 


6% 


3 


6% 


• From my modeling 


7 


9% 


3 


6% 


• Leads to better discussion 


6 


8% 


1 


2% 


• Builds confidence/self concept 


0 


0% 


2 


5% 


• NA because they don't read to students 


5 


6% 


2 


5% 


• Not responsive to question. 


1 


1% 


1 


2% 


Totals for Question 13 


77 


100% 


47 


100% 












14. Do your students read aloud during class? 










• Yes 


57 


89% 


44 


85% 


• No 


7 


11% 


8 


15% 


Totals for Question 14 


64 


100% 


52 


100% 
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Table 7. (F). Teacher Interview Results by High and Low Scoring Schools (Qualitative Data Analysis) 



Question 


High Scoring 
Schools 


Low Scoring 
Schools 




N 


% 


N 


% 


15. Please describe the various ways you use this practice. 










• Response indicates that text is read aloud in a routine manner to cover material. 


16 


23% 


10 


21% 


• Procedural texts/directions/problems/labs 


14 


20% 


8 


18% 


• Students are given time to prepare prior to reading aloud. 


6 


9% 


5 


10% 


• Students volunteer 


8 


11% 


4 


8% 


• Teacher calls on students 


7 


10% 


2 


4% 


• In small groups/pairs 


4 


6% 


6 


13% 


• To assess reading ability 


1 


1% 


2 


4% 


• Asa support for poor readers. 


0 


0% 


3 


6% 


• Reading is done in turn-taking/round robin 


4 


6% 


3 


6% 


• Oral reading for skill development 


3 


4% 


0 


0% 


• Not responsive to the question 


7 


10% 


5 


10% 


Totals for Question 15 


70 


100% 


48 


100% 












16. How do your students benefit from this practice? 










• They get the information/understand it better/comprehension 


30 


41% 


20 


39% 


• Skill development, fluency, articulation. 


17 


23% 


10 


20% 


• To keep on track/pay attention 


5 


7% 


2 


4% 


• For pleasure/appreciation/interest in reading 


5 


7% 


2 


4% 


• Builds confidence/self-esteem 


5 


7% 


8 


15% 


• Helps them understand their writing/share their writing 


1 


1% 


1 


2% 


• NA because they don't have their students read aloud 


5 


7% 


4 


8% 


• Not responsive to the question. 


5 


7% 


4 


8% 


Totals for Question 16 


73 


100% 


51 


100% 
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Table 7. (G). Teacher Interview Results by High and Low Scoring Schools (Qualitative Data Analysis) 



Question 


High Scoring 
Schools 


Low Scoring 
Schools 


17. Is time provided for your students to read for pleasure? Describe. 










• Yes 


33 


57% 


34 


64% 


o SSR/DEAR Specific school reading time 


(8) 




(3) 




o Accelerated Reader 


(4) 




(5) 




o Can read for pleasure when work is finished 


(6) 




(9) 




o Time is provided, but specific structure is not provided 


(15) 




(16) 




• No 


25 


43% 


19 


36% 


Totals for Question 17 


58 


100% 


53 


100% 












18. What do you do while your students are reading for pleasure? 










• Teacher reads 


18 


38% 


10 


27% 


• NA because students not provided reading time. 


12 


26% 


9 


24% 


• Teacher reads sometimes 


6 


13% 


3 


8% 


• Teacher grades or performs other teaching tasks 


7 


15% 


6 


16% 


• Teacher monitors students' reading 


1 


2% 


5 


14% 


• Teacher works with individual students 


2 


4% 


3 


8% 


• Not responsive to the question. 


1 


2% 


1 


3% 


Totals for Question 18 


47 


100% 


37 


100% 












19. What are some things you read? 










• Biographies/fiction 


35 


42% 


23 


36% 


• Professional literature, journals, books 


19 


22% 


20 


32% 


• Non-fiction 


11 


13% 


4 


6% 


• Anything/everything/indication they are avid readers. 


9 


11% 


7 


11% 


• Materials/novels that students would/could read 


9 


11% 


6 


9% 


• Bible 


0 


0% 


2 


3% 


• Do not read/no time to read 


1 


1% 


2 


3% 


Totals for Question 19 


84 


100% 


64 


100% 
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Table 7. (H). Teacher Interview Results by High and Low Scoring Schools (Qualitative Data Analysis) 



20. How do you use technology in your classroom? For example, Internet, 
Power Point, Web Quests, VCR, calculators, digital cameras, e-mail. 










• Internet 


42 


40% 


31 


38% 


• Moderate use/three to five types of technology mentioned 


23 


22% 


12 


15% 


• Little use/one to two types of technology mentioned 


21 


20% 


22 


27% 


• High use/many types of technology mentioned 


18 


16% 


12 


15% 


• Do not use 


2 


2% 


3 


4% 


• Not responsive to question 


0 


0% 


1 


1% 


Totals for Question 20 


106 


100% 


81 


100% 












21. How has the use of technology had an impact on your students' reading 
achievement? 










• Exposure to greater world/information/experience 


18 


31% 


11 


20% 


• Creates interest/motivation/holds attention 


16 


28% 


18 


32% 


• Increased reading achievement 


10 


18% 


2 


4% 


• Not much increase in reading achievement/don't know 


6 


10% 


4 


6% 


• Skill development has increased 


4 


7% 


9 


16% 


• It interferes with reading/library use 


2 


3% 


6 


11% 


• Has helped, students are now reading as needed 


2 


3% 


6 


11% 


Totals for Question 21 


58 


100% 


56 


100% 












22. Would you like to tell me anything else? 










• Responses were not relevant to the study. 




















23. Observations noted in classroom: Time is provided for follow-up 
discussion-Describe. 










• Yes 


34 


60% 


30 


57% 


o In-depth discussions 


(8) 




(7) 




o Limited discussions 


(1) 




(3) 




o Question and Answer session related to reading/assignments 


(10) 




(3) 




o Grouped/paired discussions among students 


(1) 




(1) 




• No 


23 


40% 


23 


43% 


Totals for Question 23 


57 


100% 


53 


100% 
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Table 7. (I) Teacher Interview Results by High and Low Scoring Schools (Qualitative Data Analysis) 



24. Observations noted in the classroom: Reading materials in classroom? 
List. 










• Yes 


52 


96% 


46 


87% 


o Supplemental texts/workbooks/dictionaries/encyclopedias 


(44) 




(41) 




o Posters/Bulletin Boards 


(30) 




(22) 




o Periodicals 


(27) 




(15) 




o Tradebooks/poetry /novels/plays 


(20) 




(25) 




o Jr. Great Books 


(3) 




(0) 




o Accelerated Reader Books 


(0) 




(2) 




• No 


2 


4% 


7 


13% 


Totals for Question 24 


54 


100% 


53 


100% 












25. Observations noted in the classroom: Quality of these materials: 
High Average Poor 










• High 


15 


26% 


9 


17% 


• High Average 


2 


3% 


3 


5% 


• Average 


38 


65% 


31 


57% 


• Low Average 


0 


0% 


1 


2% 


• Poor 


2 


3% 


5 


9% 


• Don't know 


0 


0% 


2 


4% 


• Non-existent, no reading materials 


0 


0% 


1 


2% 


• Not responsive to question 


2 


3% 


2 


4% 


Totals for Question 25 


59 


100% 


54 


100% 
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Comprehensive School Improvement Plan (CSIP) 

Data collection and analysis for CSIP 

The Comprehensive School Improvement Plans (CSIP) were obtained from the schools 
in the study. The researchers used the Comprehensive School Improvement Plan Instrument ( see 
Appendix) to conduct a content analysis of each school's CSIP. Qualitative data resulted from the 
content analysis. The data were entered into a Microsoft Word database, sorted by High Scoring 
and Low Scoring Schools, and then summarized by frequencies and percentages. 

Results for CSIP analysis. 

The results of the content analysis of the CSIP documents are presented in Table 8. 
Another version of this Table, complete with the listing of the statements in the CSIP documents 
can be found in the Appendix. The Comprehensive School Improvement Plans are designed to 
indicate how each school will address areas of weakness identified by test scores. The CSIPs 
analyzed in this study were written by school committees in response to the KCCT 2002 test 
scores. The researchers for this study reviewed only the literacy goals mentioned in the CSIP 
documents for 38 schools in the study. Thus, it might be expected that the schools with lower 
reading KCCT scores would have stronger strategies identified in the CSIP documents. An 
analysis of the data presented in Table 8 shows that this was not, in general, the case. Item 5, "Is 
there evidence of Professional Development to improve support for reading," 67% of the CSIPs 
for the high scoring schools had evidence, 79% of the low scoring schools had evidence. It seems 
logical that the low scoring schools would plan more professional development in reading. 
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Table 8. Comprehensive School Improvement Plan Content Analysis by School Reading 
Achievement Levels 



Comprehensive School Improvement Plan Item 


High < 
School 


Scoring 
'v N=21 


Low Scoring 
Schools N=14 


"No" 


"Yes" 


"No" 


"Yes" 


1. Evidence that all teachers use reading strategies across the 
curriculum. 


67% 


33% 


71% 


29% 


2. Examples of strategies used to improve reading across content area 
identified. 


57% 


43% 


64% 


36% 


3. Research-based instructional strategies are used. 


52% 


48% 


62% 


38% 


4. Content area reading strategies arc monitored in some way. 


67% 


33% 


64% 


36% 


5. Is there evidence of Professional Development to improve support 
for reading? 


33% 


67% 


21% 


79% 


6. Does the schedule provide time for independent reading? 


81% 


19% 


71% 


29% 


7. Support materials arc provided for reading instruction. 


48% 


52% 


29% 


71% 


8 Intervention for students of low reading ability is included. 


33% 


67% 


43% 


57% 



Summary of Results 
Findings. 

The two research questions that drove the study were: 

1. Do teaching practices in schools with high reading achievement scores differ from 
teaching practices in schools with low reading achievement scores? 

2. Are there differences in the inclusion of literacy goals in the Comprehensive School 
Improvement Plan between schools with high reading achievement scores and schools 
with low reading achievement scores? 

Question one. There were differences found in teaching practices in schools with high 
reading achievement scores teaching practices in schools with low reading scores. Only one 
statistically significant (p<. 05) difference was found by the Teacher Survey. Item 17: "I use 
grouping (pairs to small groups) successfully to engage students in learning." On this item, the 
mean rating for the High Scoring Schools (M=4.01, SD= 0.35) was statistically significantly 
higher than the mean rating for the Low Scoring Schools (M= 3.59, SD =0.5 1 ) [/ (36)=2.793, 



p<.05]. The means for both groups were above the scale midpoint of 3.00. 
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However, comparing this "self-report" survey data with the Teacher Observation data in 
Table 6, Item 15 should be noted for comparison. Grouping was observed in only 38% of the 
High Scoring Schools and 35% of the Low Scoring Schools. This can be accounted for by 
considering the frequency of the observations in the classroom. That is, the limitation of one day 
of observation per classroom. 

Additionally, from the Teacher Obserwition section of the study, there were two 
statistically significant (pc. 05) differences between the observed frequencies of teacher practices 
in the High Scoring Schools and the Low Scoring Schools. However, these differences were in 
an unexpected direction. On both items, the teaching practices were observed more often in the 
Low Scoring Schools. Item 1, "Teacher takes time to develop vocabulary at the beginning of the 
lesson," was observed more frequently (76%) in the Low Scoring Schools than in the High 
Scoring Schools (43%) [A 2 (l)=4.35,pc.05]. Additionally, Item 16, "Teacher engages students in 
using context clues for the vocabulary words at some point during the lesson" was observed 
statistically significantly (p<.05) more frequently in Low Scoring Schools (65%) than in High 
Scoring Schools (24%) [A 2 (l)=6.45, p<.05]. 

In general, in the Teacher Interview data (Table 7) the "flatness" of the data, that is, the 
lack of difference in measures between high achieving and low achieving schools was the most 
remarkable observation. In response to Question 4 (Table 7 B), "Tell me about some specific 
strategies you use in your classroom," 51% high scoring schools and 65% of low scoring schools had 
responses that were not descriptive of strategies. In most cases, they were descriptive of best teaching 
practices, for example, "hands-on activities," but did not fit the definition of reading strategies. However, 
teachers in 32% of high scoring schools and 2 1 % of low scoring schools identified specific strategies that 



they used. 
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In the following question (Question 5, Table 7) "Which strategies do you observe your students 
using independently," 64% of high scoring schools and 68% of low scoring schools had responses that 
were not strategies. Teachers in 17% of the high scoring schools and 21% of the low scoring schools 
mentioned specific reading strategies. On Question 10, Table 7, "Do you read aloud to your students?" 
91% of teachers in high scoring schools answered either "Yes" or "Sometimes": 95% of teachers in low 
scoring schools answered either "Yes" or "Sometimes." Responses to Question 11, "What do you read to 
them?" were varied. The most frequent response for both categories of schools was "Textbook," (26% and 
32%). 

Question two. There were few differences apparent in the Comprehensive School Improvement 
Plans of High Scoring and Low Scoring Schools (Table 8). The "Evidence that all teachers use reading 
strategies across the curriculum," was present in the CSIPs of only 33% or the High Scoring Schools and 
29% of the Low Scoring Schools. "Examples of strategies used to improve reading across content area 
identified," in 43% of the High Scoring Schools and 36% of the Low Scoring Schools. The plans for 
monitoring the content area reading strategies was (slightly) more frequently mentioned in the Low 
Scoring Schools CSIPs (36%) than in the those of the High Scoring Schools (33%). 

Professional development to improve support for reading instruction was higher in Low 
Scoring Schools, however, many of the CSIPs indicated that professional development was to be 
for English/Language Arts teachers rather than for all teachers in the schools. The Low Scoring 
Schools mentioned "Support materials are provided for reading instruction," more frequently (71%) than 
did the CSIPs of High Scoring Schools (52%). Sixty-seven percent of the High Scoring Schools 
planned to provide intervention for students of low reading ability while 57% of the Low Scoring 
Schools planned intervention for students of low reading ability. Low Scoring Schools may be 
focusing on increasing the reading ability of all of their students because their overall scores are 



so much lower. 
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Additional findings. 

As the final report for this study was being written, the researchers decided to look at the 
individual "High Scoring" and "Low Scoring" (determined by the 2002 KCCT 10th Grade 
Reading scores) high school scores on the KCCT 2003 10th Grade Reading test. Table 9 presents 
the school code, the group category for the school (high/low), and the numerical change from the 
2002 to the 2003 score. 

Table 9. Change in KCCT 10th Grade Reading Scores by Schools Within Categories 



(N=20 High Schools) 





Category Based on 2002 


Change from 2002 to 2003 




KCCT 10th Grade 


KCCT 10th Grade 


School Code 


Reading Scores 


Reading Scores 


9290 


High 


-5.8305* 


9590 


High 


3.2795 


9790 


High 


-5.1651* 


5330 


High 


12.0759 


490 


High 


10.7705 


5130 


High 


-5.2591* 


3640 


High 


3.4339 


3240 


High 


5.2107 


3540 


High 


10.6661 


90 


High 


13.0938 


190 


Low 


12.2326 


9090 


Low 


7.1087 


9490 


Low 


28.9004 


390 


Low 


7.7358 


3440 


Low 


-3.8882** 


5230 


Low 


9.5153 


5030 


Low 


2.4338 


3340 


Low 


5.8634 


9390 


Low 


19.7156 


3040 


Low 


13.3749 



Note: * Schools in the High Scoring Group with a decrease from 2002 to 2003 
**Schools in the Low Scoring Group with a decrease from 2002 to 2003 

Table 9 is presented in this special section identified as "Additional Findings " as it may 

help to explain why there were not large differences in the use of strategies used or large 



differences in the 
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Comprehensive School Improvement Plans. The underlying assumption for choosing the sample 
for this study was that High Scoring and Low Scoring schools had "status" differences in the way 
they taught reading. On the whole, these differences were not apparent from the design. 

However, the school level data for this study could easily be re-coded and sorted by "change" 
values. 
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